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ABSTRACT 



EQUAL (Educational Quality or Adult Literacy) is 
Pennsylvania's program improvement initiative that measures performance by 
the quality of education learners receive. For 3 years, Project EQUAL was 
piloted in adult education agencies throughout the state. Beginning in 1997, 
the pilot projects helped to develop a process for continuous program 
improvement for 22 adult basic and literacy education programs. During the 
first 2 years, the pilot programs systematically collected learner assessment 
data on a sample of learners. In the third year, the pilot projects reviewed 
the results of their learner assessment data and began to analyze their own 
assessment practices and results. Data were used to view program performance 
against indicators of program quality, especially in the areas of assessment, 
curriculum, and instruction. Ac the state level, assessment data submitted on 
1,800 adult learners were analyzed. After 3 years of EQUAL, many changes are 
evident, including the following: (1) programs are standardizing their 

intake, assessment, and close-out procedures; (2) new monitoring practices 
are in place to document the progress of students who leave before completing 
a program; (3) multi -service agencies are working as a team, using common 
intake and data collection procedures; (4) staff are using data collection 
and analysis as tools for program improvement, not just program requirements; 
and (5) program improvement activities are becoming part of ongoing staff 
de ve 1 opmen t . ( KC ) 
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Working for quality. 




I =iV -QUAL — Educational Quality for Adult Literacy — is Pennsylvania's program 

in 1 " 

raiA.iit 



Tl FT improvement initiative. It is transforming Pennsylvania's adult basic and 



-literacy education system from one that has judged performance by 
counting the number of people it serves to one that measures performance 
by the quality of education its learners receive. It is changing adult education with the 
help of teachers, volunteers, learners, and program administrators who are identifying 
what they need to improve teaching and learning. 



For three years Project EQUAL was piloted in adult education agencies throughout 
Pennsylvania. The pilots helped to develop a process for continuous program improve- 
ment that worked for all types of agencies, regardless of size or geographic location. 



EQUAL agencies are working in effective 
staff teams. They are conducting reliable 
and valid learner assessment. They are 
using data to understand their agencies' 
problems and identify steps to improve 
them. And they have the ability to report 
reliable, valid data to the state as well as 
use it to manage their local programs. 

What does it take to be an EQUAL 
agency? 

It takes a commitment by administrators 
and everyone who participates ... and the 
flexibility to do things differently. For 
EQUAL agencies, operations are no longer 
"business as usual" but a way of working 
that continuously improves the effective- 
ness of services to adult learners. 




We are proud in Pennsylvania to be effecting 
real change in our adult education system. 
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How EQUAL unfolded over the pilot phase 



m m m 




wenty-two adult basic and literacy education 
programs across Pennsylvania served as EQUAL pilot 
sites, beginning in 1994. They represented the 
range of providers: volunteer-based literacy councils, 
school districts, Intermediate Units, community-based organiza- 
tions, community colleges, and correctional institutions. They were 
big, small, and medium-sized, located in rural, urban, and subur- 
ban settings. Each pilot site designated a Coordinator and an 
Instructor to participate in EQUAL training activities. The EQUAL 
Advisory Board offered their expertise to the process. 



During Years One and Two, the pilots systematically collected 
learner assessment data on a sample of learners. All forms of 
assessment practices were accommodated — a variety of standard- 
ized tests, along with qualitative assessments. Coordinators 
documented and submitted sample data to the EQUAL Project 
Director. The pilots began using their data to understand self- 
selected problems in areas of concern to their local programs. 



In Year Three, the pilots reviewed the results of their learner 
assessment data and began to analyze their own assessment 
practices and results. They began more directed use of data to 
view program performance against the Indicators of Program 
Quality, especially in the areas of assessment, curriculum, and 
instruction. They used the new Program Self-Assessment Guide to 
identify areas of concern and write program improvement plans. 



At the state level, assessment data submitted on 1 ,800 adult 
learners was analyzed. Information submitted on standardized 
assessments was used to set draft Program Performance 
Standards — in other words, to help set expectations of how much 
learning can realistically be expected in a given amount of time 
based on specific measurements. The state's data reporting system 
was revised accordingly. 
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As the time to "take EQUAL to scale" 
approached, Pennsylvania's six regional 




Professional Development Centers began 
participating in EQUAL meetings. The 
purpose: to identify key training and 
technical-assistance activities needed to 
engage all state-fynded programs in 
EQUAL. Training modules were developed 



After three years of EQUAL, many 

changes are evident. Among them: 

• Programs are standardizing their intake, assessment, and 
close-out procedures. 

• New monitoring practices are in place to document the progress 
of students who leave before completing a program. 

• Multi-service agencies are working as a team, using common 
intake and data-collection procedures. 

• Staff are using data collection and analysis as tools for program 
improvement, not just program requirements. 

• Data is being made more accessible to instructors, not stuffed 
away in a file drawer. 

• Instructors — paid and volunteer — are expanding their roles as 
teachers to program planners. 

• Staff in all phases of a program are working as a team with the 
common goal of program improvement. 

• Program improvement activities are becoming a part of ongo- 
ing staff development. 



to help the PDCs deliver consistent, 
quality professional development opportu- 
nities throughout the state. 




EQUAL'S conscientiously developed program improvement model is 
now being applied in an ever larger number of programs. The 
ultimate goal: a quality basic education for all adults. 




At the core: viewing the program as a system. 




_or many programs, the first step in EQUAL — that is, 
the first step toward program improvement — is to 
.begin viewing the entire adult education program as 
an integrated system. That system is woven of many 



key elements, from how staff are hired to the agency's public 
image. At the core of the system is the connection between 
what the adult learners who walk in the door need to learn 
and how services are delivered to meet that need. 
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EQUAL provides a structure for viewing any 

adult education program as an integrated 

system ... 

• It lays a framework for determining 
individuals' needs through reliable, valid 
assessment. 

• It applies what the assessment reveals 
directly to instructional practice. 

• It ties curriculum and instruction to learner 
outcomes. 

• It produces data upon which administrative 
and educational decisions can be based. 

• It builds a team of involved staff members who work 
together toward specific ways to improve the program. 

• It provides a tested way for a program to evaluate its own 
performance. 



Student recruitment, retention, assessment, instruction, professional 
development ... these are not isolated elements but facets of a 
program that work in concert to make it what it is. The more 
effectively all the elements work together, the more effective the 
improvement process will be. 
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Teachers used to made up them own in-dass evaluations, /\fow we 
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can ready, see what is happening in the program and what factors 
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Programs are using data to make better decisions. 



I , A, 
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growing number of adult 
basic and literacy education 
programs in Pennsylvania 
are becoming involved in 
program improvement activities. They are 
making decisions and plans based on 
data, not just "gut feelings," as one 
EQUAL pilot site coordinator put it. 
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Identify a question or area of concern 



The process starts with a close look at 
each area of the program. Using the 
Agency Self-Assessment Guide, programs 
ask themselves questions related to 
Pennsylvania's Indicators of Program 
Quality. For example, Indicator #1 ad- 
dresses learners' educational gains. The 
corresponding section of the Agency Self- 
Assessment asks about the agency's 

intake and assessment process — how learner goals are determined, 
how entering students' basic skills are assessed (and reassessed), 
how information is communicated to students, how student 
progress is tracked, and how the process could be improved. 



Collect data 

▼ 

Analyze data 

: _ zx:t i 



Develop a program improvement plan 



The Data for Decision Making Log guides the program through a 
process for addressing the area of concern identified during the 
Agency Self-Assessment. What data is currently available to shed 
light on that area? What other data is needed? What would be the 
best way to collect it? What does the collected data show? Based 
on the data, the program then draws up an action plan to address 
the stated question or concern. 



For example, one adult educator reported concerns about the 
number of adult students dropping out of classes before they 




realized that a dot offiata on the, students were, in the centrad 
office and were not access/hde, to instructors, kle coidd he patting 
more assessment data in the cdassroom so teachers can use it in 

jj 

Bonnie l/adert] Adtoona Be hood District 





accomplished their goals. As a Project EQUAL exercise, the staff 
decided to investigate why. They discovered the program was not 
providing the kind of consistent feedback students need to persist 
at their education. They discovered some individuals quit the 
program because they believed they were not making sufficient 
progress. When the program improved the way it gave learners 
feedback, it dramatically reduced the number of drop-outs and 
increased the number who completed successfully. 

Some agencies work on the entire data for decision making process 
as a team. In other agencies, an administrator identifies the area of 
concern and a team completes the Data for Decision Making Log 
together. Some teachers function as leaders, completing logs and 
picking areas of concern, getting input from each other along the 
way in a very workable collaborative approach. 




Participants in the EQUAL pilot noted a 
significant improvement in staff interac- 
tion. At the beginning, some sites carried 
out specific team-building activities, which 
paid off. At the end of the three years, 
site coordinators remarked at how EQUAL 
provides a structured way for information 
to spread among staff members. They also 
noted that staff discuss issues more 
frequently and openly, are more respon- 
sible for accountability, and that a "more 
professional atmosphere" permeates the 
program. 
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Quality data use is connecting program 
improvement to performance standards. 




s the framework of a new 
statewide evaluation 
system, EQUAL has enabled 
the Pennsylvania Depart- 
ment of Education to set Program Perfor- 
mance Standards that focus on the 
effectiveness of program operations and 
adult learner progress. Assessment data 
are being used to align statewide policy 
with program improvement needs. 

Refining learner assessment practices is 
essential to the process. After all, solid 

decisions need to be based on solid information. EQUAL agencies are trained in proper 
assessment procedures to promote uniformity and consistency for all students, thus 
improving the quality of data collected. 




One program reported that they continued to use the same standardized assessment 
instrument as always but changed the timing of test administration. Before, they tested 
everyone on certain dates. Now they individually schedule testing so that enough time has 
passed between pre- and post-testing. Learning gains have become more apparent as a 
result. 



Another EQUAL pilot site analyzed its learner assessment data against the draft Program 
Performance Standards. Sure enough, they verified that their learners were demonstrating 
expected gains in all programs — even exceeding in some programs! 

In 1997-98 the Pennsylvania Department of Education is making a significant investment 
by providing federal funds to a number of agencies which have indicated a willingness to 
expand their services to new learners while participating in EQUAL training. They will be 
the first cohort of adult education agencies to analyze program and learner assessment 
data against statewide Program Performance Standards for submission to the state. 
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EQUAL pilots share their experiences ... 




t the end of the three-year 
rpy EQUAL pilot, site coordina- 



tors met for one last 
time. They shared their 



i 

experiences — the best moments and the 
challenges of their involvement in program 
improvement through data-based 
decision making. 




They talked about the 
positive changes EQUAL brought 
to their programs. They related the 
"sense of everything coming to- 
gether" as their understanding of the 
process developed. They found 
themselves asking questions about their 
programs that they did not previously have a 
structure for identifying. They pointed to the Data for Decision 
Making Logs as giving their programs direction for the future. 

In addition, pilot site coordinators were interviewed by staff of Abt 
Associates, who directed the three-year initiative. They recognized 
the importance of accurate data collection. They reported many 
new ways their programs are addressing areas of concern that 
were identified through the EQUAL process. They noted how such 
curricular, instructional, and organizational changes are affecting 
the entire program. Said one administrator, "EQUAL made us think 
about what our goals are and ask if they're being met." 



Participants in the 
three-year pilot offer 
some tips and advice 
for getting started. The 
above statements were 
actual comments of 
EQUAL pilot site 
coordinators. 



O 
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As EQUAL expands to more programs, the Bureau of Adult Basic 
and Literacy Education is providing training in learner assessment 
and the EQUAL program improvement process through the six 
regional Professional Development Centers. Many experienced 
EQUAL participants are serving as trainers. 
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For more 
information... 
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Quality improvement 
for Pennsylvania's local 
Adult Basic and 
Literacy Education 
programs is supported 
by a collaborative team 
of professionals. 

Check the chart at right 
for information on 
who is handling each 
aspect of the EQUAL 
initiative. 



BUREAU OF 
ADULT BASIC & 
LITERACY EDUCATION 




What? 



i • Program Performance-Standards 
' ♦Na mes ofJQUAL Pilot Sites ~ ZQ1. 
Names of EQUAL Advisory Board 



t • Technical Assistance 



• Agency Self-Assessment 

• Training Coordination 



Who? 



Where? 



Pa. Dept, of Education, j (717) 787-SS32 
Bureau of Adult Basic \ Fax (717) 783-0583 
& Literacy Education 



Abt Associates 



(301)718-3168 
Fax (301) 6S2-3618 



Southeastern PDC 



(717) 270-2935 



’ Assessment Training 



South-Central PDC 1 (717) 248-4942 



• Training/Professional Development 





Northwest PDC 




Central-Northeast PDC 



Southwest PDC 



South-Central PDC 



Southeastern PDC 
I0hta i i 

Philadelphia PDC 



(814) 866-3105, x223 



(814) 359-3069 



(412) 661-READ 



(717) 248-4942 



(717) 270-2935 



(21 S) 68S-6602 



PENNSYLVANIA 
DEPARTMENT OF 
EDUCATION 



333 Market St., 12th FI. 
Harrisburg, PA 17126-0333 
(717) 757-5532 




Discussing EQUAL at 
the 1997 Midwinter 
Conference on Adult 
Education , left to right: 
Sandy Strunk , SEPDC; 
Judith Alamprese , Abt 
Associates; Cheryl 
Keenan , Bureau of 
ABLE Director; Judy 
Koloski, National 
Consortium; Ella Morin 
and Sonny Sloan f Bureau 
of ABLE Division Chiefs. 
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